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the seventh left pixel in Fig. 1 9 corresponding to the third shutter time/v since the time 
of shutter opening and to the foreground component of the eighth left pixel in Fig. 19 
corresponding to the fotxth shutter time/v since the time of shutter opening. 

Since the object corresponding to the foreground is a rigid body and may be 
assumed to be moving ai: an equal speed, with the movement quantity v being 4, the 
first foreground component F02/v of the sixth left pixel in Fig. 1 9, with the first shutter 
time/v since the time of shutter opening, is equal to the second foreground component 
of the seventh left pixel in Fig. 19 corresponding to the second shutter time/v since the 
time of shutter opening. Similarly, the foreground component F02/v is equal to the 
foreground component of the eighth left pixel in Fig. 19 corresponding to the third 
shutter time/v since the time of shutter opening. 

Since the object corresponding to the foreground is a rigid body and may be 
assumed to be moving at an equal speed, with the movement quantity v being 4, the 
first foreground component F03/v of the seventh left pixel in Fig. 19, with the first 
shutter time/v since the Lime of shutter opening, is equal to the second foreground 
component of the eighth left pixel in Fig. 1 9 corresponding to the second shutter time/v 
since the time of shutter opening. 

Although the number of times of the virtual splitting is four in the description 
with respect to Figs. 17 to 19, the number of times of the virtual splitting corresponds 
to the movement quantity v. The movement quantity v generally corresponds to the 
movement speed of the object corresponding to the foreground. For example, if the 
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object corresponding to the foreground is moving so as to be displayed four pixels 
rightwards in a frame next to a previous reference frame, the movement quantity v is 
4. The number of times of the virtual splitting is set to 4 in association with the 
movement quantity v. Similarly, if the object corresponding to the foreground is 
moving so as to be displayed six pixels rightwards in a frame next to a previous 
reference frame, the movement quantity v is 6, with the number of times of the virtual 
splitting being six. 

Figs 20 and 2 1 sho w the relation between the foreground area, background area, 
and the mixed area, comprised of the covered background area and the uncovered 
background area, on one hand, and the foreground and background components 
corresponding to the spli: shutter time, on the other hand, as described above. 

Fig.20 shows an example of extraction of pixels of the foreground area, 
background area and the mixed area as extracted from a picture corresponding to an 
object moving before a still background. In the embodiment shown in Fig.20, an 
object corresponding to \ho foreground is moving horizontally with respect to the 
picture. 

The frame #n+l is a frame next to the frame #n, with the frame #n+2 being a 
frame next to the frame #n-H. 

Fig.21 diagrammalically shows a model obtained on extracting pixels of the 
foreground area, background area and the mixed area, extracted in turn from one of 
the frames #n to #n+2, with the movement quantity v being 4, and on expanding the 
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pixel values of the extracted pixels along the time axis direction. 

Since the object corresponding to the foreground is moved, the pixel values of 
the foreground area are constituted by four different foreground components 
corresponding to the period of the shutter time/v. For example, the leftmost one of 
pixels of the foreground iirea shown in Fig.2 1 are F0 1/v, F02/v, F03/v and F04/v. That 
is the pixels of the fore;*round area contain are corrupted with motion blurring. 

Since the object corresponding to the background is at a standstill, the light 
corresponding to the background input to the sensor 1 1 during the time corresponding 
to the shutter time is not changed. In this case, the pixel values of the background are 
free of the motion blurring. 

The pixel values or the pixels belonging to the mixed area composed of the 
covered background area or the uncovered background area are comprised of the 
foreground and background components. 

A model comprised of neighboring pixels in a row in plural frames, in which the 
pixel values of pixels lyin ? at the same position on a frame are developed in the time 
axis direction, with the picture corresponding to an object being moved, is explained. 
For example, if the picture corresponding to the object is moving horizontally with 
respect to the picture, the pixels arrayed on the same row on the picture may be 
selected as the pixels in a row in a picture. 

Fig.22 diagrammatically shows a model obtained on temporally expanding the 
pixel values of pixels arrayed in a row of each of three frames of a picture of a 
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photographed object corresponding to a still background, with the developed pixels 
being at the same positions on the respective frames. The frame #n is the frame next 
to the frame #n-l, with the frame #n+l being the frame next to the frame #n. The 
remaining frames are teimed in similar manner. 

The pixel values, of B01 to B12 shown in Fig. 22 are those of pixels 
corresponding to the object of the still background. Since the objet corresponding to 
the background is at a st andstill, the pixel values of the corresponding pixels in the 
frames fhr 1 ti frame n+1 are not changed. For example, the pixel in the frame #n and 
the pixel in the frame #n H, corresponding to the positions of the pixels having pixel 
values of BOS in the frame #nr 1 , are of pixel values of B05. 

Fig.23 shows pixel values of neighboring pixels in a row in each of three frames 
of a photographed picture of an object corresponding to the foreground moving 
rightwards in Fig.23, alon g with the object corresponding to the still background, with 
the pixel values being sh own developed along the time axis direction. The model 
shown in Fig.23 includes a covered background area. 

In Fig.23, the object corresponding to the foreground is a rigid body and may 
be assumed to be moving ;it a constant speed, with the foreground picture being moved 
so that the foreground picture will be displayed four pixels rightwards in the next 
frame. So, the movement quantity v of the foreground is 4, with the number of times 
of the virtual splitting being 4. 

For example, the foreground component of the leftmost pixel of the frame #tr 1 
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in Fig.23, with the first sh utter time /v since the opening of the shutter, is F12/v, whilst 
the foreground component of the second left pixel, with the second shutter time /v 
since the opening of the shutter, is also F12v. The foreground component of the third 
left pixel in Fig, 23, with the third shutter time/v since the opening of the shutter, and 
the foreground component of the fourth left pixel in Fig.23, with the fourth shutter 
time/v since the opening of the shutter, are each F12/v. 

For example, the fc reground component of the leftmost pixel of the frame #n* 1 
in Fig.23, with the second shutter time /v since the opening of the shutter, is Fl 1/v, 
whilst the foreground component of the second left pixel, with the third shutter time 
/v since the opening of the shutter, is also Fllv. The foreground component of the 
third left pixel in Fig.23, with the fourth shutter time/v since the opening of the shutter, 
is Fl 1/v. 

The foreground component of the leftmost pixel of the frame #n« 1 in Fig.23, 
with the third shutter time /v since the opening of the shutter, is FlO/v, whilst the 
foreground component of the second left pixel, with the fourth shutter time /v since 
the opening of the shutter , is also FlOv. The foreground component of the leftmost 
pixel in Fig.23, with the fourth shutter time/v since the opening of the shutter, is F09/v. 

Since the object corresponding to the background is at a standstill, the 
background component of the second left pixel of the frame #n-l in Fig.23, with the 
first shutter time/v as from the shutter opening time, is B0 1/v. The background 
component of the third left pixel of the frame #n- 1 in Fig.23, with the first and second 
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shutter time/v as from the shutter opening time, is B02/v, while the background 
component of the fourth left pixel of the frame #n-l in Fig.23, with the first to third 
shutter time/v as from tie shutter opening time, is B03/v. 

In the frame #n-l in Fig.23, the leftmost pixel belongs to the foreground area, 
while the second to fouri left pixels belong to the mixed area which is the covered 
background area. 

The fifth to twel ith left pixels of the frame #n-l in Fig.23 belong to the 
background area, with the corresponding pixel values being B04 to B 1 1, respectively. 

The first to fifth pixels of the frame #n l in Fig.23 belong to the background 
area. The foreground component in the foreground area of the frame #n, with the 
shutter time/v, is one of F05v to F12/v. 

Since the object corresponding to the foreground is a rigid body and may be 
assumed to be moving at a constant speed, with the foreground picture being moved 
so that the foreground picture will be displayed four pixels rightwards in the next 
frame, the foreground component of the fifth left pixel of the frame #n-in Fig.23, with 
the first shutter time /v since the opening of the shutter, is F12/v, whilst the foreground 
component of the sixth left pixel, with the second shutter time /v since the opening of 
the shutter, is also F12v. The foreground component of the seventh left pixel in 
Fig.23, with the third shutter time/v since the opening of the shutter, and the 
foreground component of vhe eighth left pixel in Fig.23, with the fourth shutter time/v 
since the opening of the shutter, are each F12/v. 
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The foreground component of the fifth left pixel of the frame #n in Fig.23, with 
the second shutter time /v since the opening of the shutter, is Fl 1/v, whilst the 
foreground component of the sixth left pixel, with the third shutter rime /v since the 
opening of the shutter, is also Fl lv. The foreground component of the seventh left 
pixel in Fig.23, with the fourth shutter time/v since the opening of the shutter, is Fl 1 /v. 

The foreground component of the fifth left pixel of the frame #n in Fig.23, with 
the third shutter time N since the opening of the shutter, is FlO/v, whilst the 
foreground component of the sixth left pixel, with the fourth shutter time /v since the 
opening of the shutter, is jdso FlOv. The foreground component of the fifth left pixel 
in Fig.23, with the fourth shutter time/v since the opening of the shutter, is F09/v. 

Since the object sorresponding to the background is at a standstill, the 
background component of the sixth left pixel of the frame #n in Fig.23, with the first 
shutter time/v as from the «hutter opening time, is B05/v. The background component 
of the seventh left pixel of the frame #n in Fig.23, with the first and second shutter 
time/v as from the shutter opening time, is B06/v, while the background component 
of the eighth left pixel of the frame #n in Fig.23, with the first to third shutter time/v 
as from the shutter opening time, is B07/v. 

In the frame #n- 1 in Fig.23, the first to ninth left pixels belong to the foreground 
area, while the sixth to eigh th left pixels belong to the mixed area which is the covered 
background area. 

The first to ninth to twelfth left pixels of the frame #n+l in Fig.23 belong to the 
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foreground area, with the pixel values being B08 to Bl 1, respectively. 

The first to ninth pixels of the frame #n+l in Fig.23 belong to the foreground 
area. The foreground component in the foreground area of the frame #n+l, with the 
shutter time/v, is one of FOlv to F12/v. 

Since the object corresponding to the foreground is a rigid body and may be 
assumed to be moving at a constant speed, with the foreground picture being moved 
so that the foreground picture will be displayed four pixels rightwards in the next 
frame, the foreground component of the ninth left pixel of the frame #n+l in Fig.23, 
with the first shutter time N since the opening of the shutter, is F12/v, whilst the 
foreground component of the tenth left pixel, with the second shutter time N since the 
opening of the shutter, is also F12v. The foreground component of the eleventh left . 
pixel in Fig.23, with the third shutter time/v since the opening of the shutter, and the 
foreground component of the twelfth leftpixel in Fig.23, with the fourth shutter time/v 
since the opening of the shutter, are each Fl 2/v. 

The foreground component of the ninth leftpixel of the frame #n+l in Fig.23, 
with the second shutter time N since the opening of the shutter, is Fl 1/ Vj whilst the 
foreground component of the tenth left pixel, with the third shutter time /v since the 
opening of the shutter, is also Fl Iv. The foreground component of the eleventh left 
pixel in Fig.23, with the fourth shutter time/v since the opening of the shutter, is Fl 1/v. 

The foreground component of the ninth left pixel of the frame #n+l in Fig.23, 
with the third shutter time /v since the opening of the shutter, is FlO/v, whilst the 
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foreground component c f the tenth left pixel, with the fourth shutter time /v since the 
openingofthe shutter, is alsoFlOv. The foreground component of the ninth left pixel 
of the frame #n+lin Fig.23, with the fourth shutter time/v since the opening of the 
shutter, is F09/v. 

Since the object corresponding to the background is at a standstill, the 
background component of the tenth left pixel of the frame #n+l in Fig.23, with the 
first shutter time/v as fram the shutter opening time, is B09/v. The background 
component of the eleventh left pixel of the frame #n+l in Fig.23, with the first and 
second shutter time/v as from the shutter opening time, is BlO/v, while the background 
component of the twelfth left pixel of the frame #n+l in Fig.23, with the first to third 
shutter time/v as from the shutter opening time, is B 1 1 /v. 

In the frame #n+l in Fig.23, the tenth to twelfth left pixels correspond to the 
mixed area which is the covered background area. 

Fig.24 diagrammatically shows a picture obtained on extracting the foreground 
component from the pixel values shown in Fig.23. 

Fig.25 shows neighboring pixels in a row of each of three frames of a 
photographed picture ofthe foreground corresponding to an objectmovingrightwards 
in the drawing, along with the still background. In Fig.25, there is also shown the 
uncovered background area. 

In Fig.25, the objeci: corresponding to the foreground is a rigid body and may 
be assumed to be moving at a constant speed, with the foreground picture being moved 
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so that the foreground picture will be displayed four pixels rightwards in the next 
frame. So, the movement quantity v of the foreground is 4. 

For example, the foreground component of the leftmost pixel of the frame #n* 1 
in Fig.25, with the first shutter time /v since the opening of the shutter, is F 1 3/v, whilst 
the foreground component of the second left pixel, with the second shutter time /v 
since the opening of the shutter, is also F13v. The foreground component of the third 
left pixel in Fig. 23, with the second shutter time/v since the opening of the shutter, and 
the foreground component of the fourth left pixel in Fig.25, with the fourth shutter 
time/v since the opening of the shutter, are each Fl 3/v. 

For example, the foreground component of the second left pixel of the frame 
#n- 1 in Fig.23, with the first shutter time /v since the opening of the shutter, is F14/v, 
whilst the foreground component of the third left pixel, with the second shutter time 
/v since the opening of the shutter, is also F14v. The foreground component of the 
third left pixel in Fig.25, with the first shutter time/v since the opening of the shutter, 
isF15/v. 

Since the object corresponding to the background is at a standstill, the 
background component of the leftmost pixel of the frame #n-l in Fig.25, with the 
second to fourth shutter time/v as from the shutter opening time, is BOl/v. The 
background component of the second left pixel of the frame #n*l in Fig.25, with the 
third and fourth shutter time/v as from the shutter opening time, is B26/v, while the 
background component of the third left pixel of the frame #n-l in Fig.25, with the 
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fourth shutter time/v as fiom the shutter opening time, is B27/v. 

In the frame #n- 1 ir. Fig.25, the first to third left pixel belongs to the mixed area 
which is the covered background area. 

The fourth to twelfth left pixels of the frame #n*l in Fig;25 belong to the 
foreground area, with the foreground component of the foreground of the frame being 
one of F13v to F24v. 

The first to fourth left pixels of the frame #n in Fig.25 belong to the background 
area, with the pixel values being B25 to B28, respectively. 

Since the object corresponding to the foreground is a rigid body and may be 
assumed to be moving at ;i constant speed, with the foreground picture being moved 
so that the foreground picture will be displayed four pixels rightwards in the next 
frame, the foreground component of the fifth left pixel of the frame #n in Fig. 23, with 
the first shutter time /v sirn^e the opening of the shutter, is F 1 3/v, whilst the foreground 
component of the sixth left pixel, with the second shutter time /v since the opening of 
the shutter, is also F13v. The foreground component of the seventh left pixel in 
Fig.25, with the third shutter time/v since the opening of the shutter, and the 
foreground component of the eighth left pixel in Fig.25, with the fourth shutter time/v 
since the opening of the shutter, are each F13/v. 

The foreground component of the sixth left pixel of the frame #n in Fig. 2 3 , with 
the first shutter time /v since the opening of the shutter, is F14/v, whilst the foreground 
component of the seventh left pixel, with the second shutter time /v since the opening 
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of the shutter, is also F14v. The foreground component of the eighth left pixel in 
Fig.25, with the first shutter time/v since the opening of the shutter, is F15/v. 

Since the object corresponding to the background is at a standstill, the 
background component of the fifth left pixel of the frame #n in Fig.25, with the second 
to fourth shutter time/v as from the shutter opening time, is B29/v. The background 
component of the sixth left pixel of the frame #n in Fig.25, with the third and fourth 
shutter time/v as from the shutter opening time, is B30/v, while the background 
component of the seventh left pixel of the frame #n inFig.23, with the fourth shutter 
time/v as from the shutter opening time, is B31/v. 

In the frame #n in Fig.25, the first to ninth left pixels belong to the foreground 
area, while the fifth to seventh left pixels belong to the mixed area which is the 
covered background area. 

The eighth to twelfth left pixels of the frame #n+l in Fig.25 belong to the 
foreground area, with the pixel values being B25 to B32, respectively. 

The first to eighth pixels of the frame #n+l in Fig.25 belong to the background 
area, with the pixel values being B25 to B32, respectively. 

Since the object corresponding to the foreground is a rigid body and may be 
assumed to be moving at a constant speed, with the foreground picture being moved 
so that the foreground picture will be displayed four pixels rightwards in the next 
frame, the foreground component of the ninth left pixel of the frame #n+l in Fig.25, 
with the first shutter time /v since the opening of the shutter, is F13/v, whilst the 
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foreground component of the tenth left pixel, with the second shutter time /v since the 
opening of the shutter, is also F13v. The foreground component of the eleventh left 
pixel in Fig. 25, with the third shutter time/v since the opening of the shutter, and the 
foreground component o:f the twelfth left pixel in Fig. 25, with the fourth shutter time/v 
since the opening of the shutter, are each F13/v. 

The foreground component of the tenth left pixel of the frame #n+l in Fig.25, 
with the first shutter time/v since the opening of the shutter, is F14/v, whilst the 
foreground component of the eleventh left pixel, with the second shutter time /v since 
the opening of the shutter, is also F 14v. The foreground component of the twelfth left 
pixel in Fig.25, with the first shutter time/v since the opening of the shutter, is Fl 5/v. 

Since the object corresponding to the background is at a standstill, the 
background component of the ninth left pixel of the frame #n+l in Fig.25, with the 
second to fourth shutter time/v as from the shutter opening time, is B33/v. The 
background component of the tenth left pixel of the frame #n+l in Fig.25, with the 
third and fourth shutter time/v as from the shutter opening time, is B34/v, while the 
background component of the eleventh left pixel of the frame #n+ 1 in Fig.25, with the 
fourth shutter time/v as from the shutter opening time, is B35/v. 

In the frame #n+l i n Fig.25, the ninth to eleventh left pixels correspond to the 
mixed area which is the covered background area. 

In Fig.25, the twelfth left pixel of the frame #n+I belong to the foreground area. 
The foreground component with the shutter time/v in the foreground area of frame 
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#n+l is one of F13v to F16v. 

Fig.26 diagramma Really shows a picture obtained on extracting the foreground 
component from the pixe l values shown in Fig.25. 

Reverting to Fig. 10, the area specifying unit 103 associates a flag, indicating 
that a given picture belong to the foreground area, a background area, a covered 
background area or an uncovered background area, from pixel to pixel, using pixel 
value of plural frames, ar.d routes the resulting areal information to the mixing ratio 
calculating unit 104 and to the motion blurring adjustment unit 106. 

Based on the pixel values of plural frames and the areal information, the mixing 
ratio calculating unit 104 computes the mixing ratio a for each of the pixels contained 
in the mixed area, ;md sends the computed mixing ratio ct to the 
foreground/background separating unit 105. 

Based on the pixel values of the plural frames, areal information and the mixing 
ratio a, the foreground/background separating unit 105 extracts the foreground 
component picture made up only of the foreground component to send the extracted 
component picture to the motion blurring adjustment unit 106. 

Based on the foreground component picture sent from the 
foreground/background separating unit 105, the motion vector sent from the motion 
detection unit 102 and on the areal information sent from the area specifying unit 103, 
the motion blurring adjustment unit 106 adjusts the quantity of the motion blurring 
contained in the foreground component picture to output the foreground component 



PACE 15/51 ■ RCVD AT 7/16/2009 1:55:20 PM [Eastern Daylight Time) * SVR:USPTO-EFXRF-5/18 » DNIS: 2738300 * CSID:1 2125880500 ■ DURATION (mm-ss): 10-26 



07/16/2009 13:58 FAX 12125880500 



FROMMER LAWRENCE & HAUG 



©016/051 



65 

picture adjusted for the motion blurring. 

Referring to the flowchart of Fig.27, the processing for adjusting the motion 
blurring caused by the signal processor 12 is explained. At step SI 01, the area 
specifying unit 103 executes the area specifying processing for generating the areal 
information indicating to which of the foreground area, background area, covered 
background area or the ijncovered background area belong the pixels of the input 
picture, from one pixel of the input picture to another. The area specifying processing 
will be explained subsequently by referring to the flowchart of Fig.36. The area 
specifying unit 103 sends the generated area information to the mixing ratio 
calculating unit 104. 

Meanwhile, the area specifying unit 103 at step S101 may generate the areal 
information indicating to which of the foreground area, background area or the mixed 
area belong the pixels of the input picture, from one pixel of the input picture to 
another, based on the input picture. In this case, no distinction is made between the 
covered background area and the uncovered background area. In this case, the 
foreground/background separating unit 105 and the motion blurring adjustment unit 
106 decide whether the mixed area is the covered background area or the uncovered 
background area, based on the direction of the motion vector. For example, if the 
foreground area, mixed area and the background area are arrayed sequentially in 
association with the direction of the motion vector, the mixed area is verified to be the 
covered background area, whereas, if the background area, mixed area and the 
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foreground area are arrayed sequentially in association with the direction of the motion 
vector, the mixed area is verified to be the uncovered background area. 

At step S 102, the mixing ratio calculating unit 104 calculates the mixing ratio 
a, from one pixel contained in the mixing area to another, based on the input picture 
and the area information The processing for computing the mixing ratio will be 
explained in detail subsequently by referring to the flowchart of Fig.46. The mixing 
ratio calculating unit 104 sends the computed mixing ratio a to the 
foreground/background stsparating unit 105. 

At step S103, the foreground/background separating unit 105 extracts the 
foreground component from the input picture, based on the motion vector and the areal 
information, to send the extracted component to the motion blurring adjustment unit 
106 as the foreground component picture. 

At step S104, the m otion blurring adjustment unit 106 generates a processing 
unit for indicating a position on the picture of pixels arrayed consecutively in the 
movement direction of each of the uncovered background area, foreground area and 
the covered background area, based on the motion vector and on the area information, 
to adjust the quantity of the motion blurring contained in the foreground component 
corresponding to the processing unit The processing for adjusting the quality of the 
motion blurring will be explained subsequently by referring to the flowchart of Fig.63. 

At step S 1 05, the signal processor 12 verifies whether ornot the processing has 
been finished for the entire picture. If the signal processor 12 has verified that the 
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processing has not been finished fort the entire picture, it proceeds to step SI 04 to 
repeat the processing for adjusting the quantity of the motion blurring for the 
foreground component corresponding to the processing unit. 

If, at step S 106, it i:j verified that the processing has been finished for the entire 
picture, the processing is terminated. 

In this manner, the signal processor 12 is able to separate the foreground and 
the background from each other to adjust the quantity of the motion blurring contained 
in the foreground. That is, the signal processor 12 is able to adjust the amount of 
motion blurring contained in sample data as pixel value of the foreground pixel. 

In the following, illustrative structures of the area specifying unit 103, mixing 
ratio calculating unit 104, foreground/background separating unit 105 and the motion 
blurring adjustment unit 106 are hereinafter explained. 

Fig. 2 8 is a block dia;$ram showing an illustrative structure of the area specifying 
unit 103. A frame memory 121 stores an input picture on the frame basis. When a 
frame being processed is a frame #n, the frame memory 121 stores a frame #n"2, as 
a frame two frames before the frame #n, a frame #n* 1, as a frame one frame before 
the frame #n, a frame #n+l, as a frame one frame after the frame #n, and a frame 
#n+2, as a frame two frames after the frame #n. 

A still/movement discriminating unit 122-1 reads out a pixel value of a pixel 
of the frame #n+2 lying at ihe same position as the position on the picture of the pixel 
of the frame #n being area specified, and a pixel value of a pixel of the frame #n+l 
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lying at the same positior. as the position on the picture of the pixel of the frame #n 
being area-specified, from, the frame memory 12 1, to calculate an absolute value of the 
difference of the read-our pixel values. The still/movement discriminating unit 122-1 
verifies whether or not the absolute value of the difference between the pixel value of 
the frame #n+2 and the frame #n+l is larger than a predetermined threshold value Th. 
If it is verified that the absolute value of the difference is larger than the threshold 
value Th, the still/movement discriminating unit 122-1 routes a still/movement 
decision specifying the movement decision to an area decision unit 123-1. If it is 
verified that the absolute value o the difference between the pixel value of the frame 
#n+2 nd the pixel value of the frame #n+l is not larger than the threshold value Th, 
the still/movement discriminating unit 122-1 routes a still/movement decision 
specifying the still decision to an area decision unit 123-1. 

A still/movement discriminating unit 122-2 reads out a pixel value of a pixel of 
the frame #n+l lying at the same position as the position on the picture of the pixel of 
the frame #n being area-specified, and a pixel value of a pixel of the frame #n+l lying 
at the same position as the position on the picture of the pixel of the frame #n being 
area-specified, from the frame memory 121, to calculate an absolute value of the 
difference of the read-our pixel values. The still/movement discriminating unit 122-2 
verifies whether or not the absolute value of the difference between the pixel value of 
the frame #n+l and the frame #n is larger than a predetermined threshold value Th. 
If it is verified that the absolute value of the difference between the is larger than the 
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threshold value Th, the still/movement discriminating unit 122-1 routes a 
still/movement decision specifying the movement decision to an area decision unit 
123-land to an area decision unit 123-2. If it is verified that the absolute value of the 
difference between the pixel value of the pixel of the frame #n-H and that of the pixel 
of the frame #n is net larger than the threshold value Th, the still/movement 
discriminatingunit 122- 1 routes a still/movement decision specifying the still decision 
to an area decision unit 123-1 and to an area decision unit 123-2. 

A still/movement discriminating unit 122-3 reads out a pixel value of apixel of 
the frame #n lying at the same position as the position on the picture of the pixel of the 
frame #n being area-specified, and a pixel value of a pixel of the frame #n-l lying at 
the same position as the position on the picture of the pixel of the frame #n being area- 
specified, from the frame memory 121, to calculate an absolute value of the difference 
of the read-our pixel values. The still/movement discrinunating unit 122-3 verifies 
whether or not the absolute value of the difference between the pixel value of the 
frame #n and the frame tor 1 is larger than a predetermined threshold value Th. If it 
is verified that the absohile value of the difference between the pixel values is larger 
than the threshold value lb, the still/movement discriminating unit 122-3 routes a 
still/movement decision specifying the movement decision to an area decision unit 
123-land to an area decision unit 123-3. If it is verified that the absolute value of the 
difference between the pixel value of the pixel of the frame #n and that of the pixel of 
the frame #n-l is not larger than the threshold value Th, the still/movement 
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discriminating unit 1 routes a still/movement decision specifying the still decision 
to an area decision unit 5 23-2 and to an area decision unit 123-3. 

A still/movement discriminating unit 1 22-4 reads out the pixel value of the pixel 
of the frame #n- 1 lying ai the same position as the position on the picture of the pixel 
of the frame #*n being are a-specified, and the pixel value of the pixel of the frame #n-2 
lying at the same position on the picture of the pixel of the frame #n being area- 
specified, to calculate ths absolute value of the difference of the pixel values. The 
still/movement discriminating unit 122-4 verifies whether or not the absolute value of 
the difference of the pixel value of the frame #n-l and the pixel value of the frame 
#n-2 is larger than the predetermined threshold value Th. If the absolute value of the 
difference between the pixel value of the frame #n- 1 and the pixel value of the frame 
#n-2 is verified to be larger than the threshold value Th, a still/movement decision 
indicating the decision for movement is routed to the area decision unit 123-3. If it is 
verified that the absolute value of the difference between the pixel value of the frame 
#n- 1 and the pixel value of the frame #n-2 is not larger than the threshold value Th, the 
still/movement discriminating unit 122-4 routes a still/movement decision indicating 
the still decision to the area decision unit 123-3. 

If the still/movement decision routed from the still/movement discriminating 
unit 122-1 indicates still and the still/movement decision routed from the 
still/movement discriminating unit 122-2 indicates movement, the area decision unit 
123-1 decides that the pixel on the frame #n being area-specified belongs to the 
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uncovered background ;jrea and sets "1" in an uncovered background area decision 
flag associated with the pixel being area-specified for indicating that the pixel belongs 
to the uncovered background area. 

If the still/movement decision routed from the still/movement discriminating 
unit 122-1 indicates movement and the still/movement decision routed from the 
still/movement discrinimating unit 122-2 indicates still, the area decision unit 123-1 
decides that the pixel on the frame #n being area-specified does not belong to the 
uncovered background area and sets "0" in an uncovered background area decision 
flag associated with the p,xel being area-specified for indicating that the pixel does not 
belong to the uncovered background area 

The area decision unit 123-1 routes the uncovered background area decision 
flag, having "1" or "0" set in this manner, to a decision flag storage memory 124. 

If the still/movement decision routed from the still/movement discriminating 
unit 122-2 indicates slill and the still/movement decision routed from the 
still/movement discrinumiting unit 122-3 indicates still, the area decision unit 123-2 
decides that the pixel on the frame #n being area-specified belongs to the still area and 
sets "1" in a still area decision flag associated with the pixel being area-specified for 
indicating that the pixel belongs to the uncovered background area. 

If Hie still/movement decision routed from the still/movement discriminating 
unit 122-2 indicates movement or the still/movement decision routed from the 
still/movement discriminating unit 122-3 indicates movement, the area decision unit 
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123-2 decides that the pixel on the frame #n being area-specified does not belong to 
the still area and sets "0" in a still area decision flag associated with the pixel being 
area-specified for indica ting that the pixel does not belong to the still area. 

The area decision unit 123-2 routes the still area decision flag, thus having "1" 
or "0" set therein, to the decision flag storage memory 124. 

If the still/movement decision routed from the still/movement discriminating 
unit 122-2 indicates movement and the still/movement decision routed from the 
still/movement discrimin ating unit 122-3 indicates movement, the area decision unit 
123-2 decides that the pixel on the frame #n being area-specified belongs to the 
movement area and sets " 1" in a movement area decision flag associated with the pixel 
being area-specified for indicating that the pixel belongs to the movement area. 

If the still/movement decision routed from the still/movement discriminating 
unit 122-2 indicates still orthe still/movement decision routed from the still/movement 
discriminating unit 122-3 indicates still, the area decision unit 123-2 decides that the 
pixel on the frame #n being area-specified does not belong to the movement area and 
sets "0" in a movement area decision flag associated with the pixel being area- 
specified for indicating that the pixel does not belong to the movement area. 

The area decision unit 123-2 routes the movement area decision flag, thus 
having "1" or "0" set therein, to the decision flag storage memory 124. 

If the still/movement decision routed from the still/movement discriminating 
unit 122-3 indicates movement and the still/movement decision routed from the 
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still/movement discrimirating unit 122-4 indicates still, the area decision unit 123-3 
decides that the pixel on the frame #n being area-specified belongs to the uncovered 
background area and sets "1 " in a covered background area decision flag associated 
with the pixel being area-specified for indicating that the pixel belongs to the covered 
background area. 

If the still/movement decision routed from the still/movement discriminating 
unit 1 22-3 indicates still or the still/movement decision routed from the still/movement 
discriminating unit 122-4 indicates movement, the area decision unit 123-3 decides 
that the pixel on the frame #n being area-specified does not belong to the covered 
background area and sets "0" in a covered background area decision flag associated 
with the pixel being area-specified for indicating that the pixel does not belong to the 
covered background area. 

The area decision unit 123-3 routes the covered background area decision flag, 
thus having " 1" or "0" set ilierein, to the covered background area decision flag storage 
memory 124. 

The decision flag storage memory 124 stores the uncovered background area 
decision flag, sent from the area decision unit 123-1, the still area decision flag, sent 
from the area decision unit 123-2, the movement area decision flag, sent from the area 
decision unit 123-2, and the uncovered background area decision flag, sent from the 
area decision unit 123-3. 

The decision flag storage memory 124 sends the uncovered background area 
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decision flag, still area decision flag, movement area decision flag and the covered 
background area decision flag to a synthesis unit 125. Based on the uncovered 
background area decision flag, still area decision flag, movement area decision flag 
and the covered background area decision flag, supplied from the decision flag storage 
memory 124, the synthesis unit generates the area information indicating to which of 
the uncovered backgroun d area, still area, movement area and the covered background 
area belong the respective; pixels, and routes the information so generated to a decision 
flag storage frame memory 126. 

The decision flag storage frame memory 126 stores the area information, 
supplied from the synthesis unit 125, while outputting the area information stored 
therein. 

Referring to Figs.29 to 33, a typical processing by the area specifying unit 1 03 
is explained. 

When an object corresponding to the foreground is moving, the position of the 
picture corresponding to the object on the picture screen is changed from frame to 
frame. Referring to Fig.29, a picture corresponding to an object at a position Yn(x, y) 
in a frame #n is positioned at Yn+l(x,y) at the next frame #n+l . 

Fig. 30 diagrammal ically shows a model of pixel values of a row of pixels 
neighboring to one another along the moving direction of the picture corresponding 
to the foreground. For example, if the movement direction of the picture 
corresponding to the foreground is horizontal relative to the picture screen, the 
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diagrammatic view of Fig.30 shows a model in which pixel values of pixels 
neighboring to one another on one line are developed in the time axis direction. 
In Fig.30, the line in the frame #n is the same as one in the frame #n+l . 
The components of the foreground corresponding to the object contained in the 
second to the thirteenth pixels as counted from left in the frame #n are included in the 
sixth to seventeenth pixels as counted from the left of the frame #n+l . 

The pixels belon^g to the covered background area in the frame Un are the 
eleventh to thirteenth pixels as counted from left, whilst the pixels belonging to the 
uncovered background area are the second to fourth pixels as counted from left. The 
pixels belonging to the covered background area in the frame #n+I are the fifteenth 
to seventeenth pixels as counted from left, whilst the pixels belonging to the uncovered 
background area are the sixth to eighth pixels as counted from left. 

In the example shown in Fig.30, since the foreground component in the frame 
#n are moved by four pixels in the frame #n+l, the movement quantity v is 4. The 
number of times of the virtual splitting corresponds to the movement quantity and is 
equal to 4. 

The change in the pixel values of pixels belonging to the mixed area ahead and 
at back of the frame being considered is explained. 

In the frame #n shown in Fig.31, in which the background is still and the 
movement quantity of the foreground v is 4, pixels belonging to the covered 
background area are fifteeDth to seventeenth pixels from left. Since the movement 
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quantity v is 4, the fifteenth to seventeenth pixels from left in the directly previous 
frame #n- 1 contain only the background components and belong to the background. 
The fifteenth to seventeenth pixels from left in the further previous frame #n*2 contain 
only the background components and belong to the background area. 

Since the object corresponding to the background is still, the pixel value of the 
fifteenth pixel from the left of the frame #n-l is not changed from the pixel value of 
the fifteenth pixel from the left of the frame ftr2. Similarly, the pixel value of the 
sixteenth pixel from the left of the frame #n* 1 is not changed from the pixel value of 
the sixteenth pixel from the left of the frame #n-2, whilst the pixel value of the 
seventeenth pixel from the left of the frame #n- 1 is not changed from the pixel value 
of the seventeenth pixel from the left of the frame #n*2. 

That is, the pixels of the frame fori and the frame #n*2 corresponding to the 
pixels belonging to the covered background area in the frame #n are comprised only 
of the background components and are not changed, so that the absolute value of the 
difference is substantially 0. So, the still/movement decision on the pixels of the frame 
#n* 1 and frame #n*2 corresponding to the mixed area in the frame #n is made as being 
still by the still/moving discriminating unit 122-4. 

Since the pixels belonging to the covered background area in the frame #n 
contain the foreground components, the corresponding pixel values differ from those 
in which the pixels are comprised only of background components in the frame #n-L 
Therefore, the pixels belonging to the mixed area in the frame #n and the 
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corresponding pixels of the frame #n*l are verified to be moving pixels by the 
still/moving discriminating unit 122-3. 

When fed with the result of still/movement decision indicating the movement 
from the still/moving dis criminating unit 122-3 and with the result of still/movement 
decision indicating the still from the still/moving discriminating unit 122-4, the area 
decision unit 123-3 decides that the pixel in question belongs to the covered 
background area. 

The pixels contain ed in the uncovered background area in the frame #n in which 
the background is still and the movement quantity v of the foreground is 4 are second 
to fourth pixels as counted from left. Since the movement quantity v is 4, the second 
to fourth pixels from left in the next frame #n+l contain only the background 
components and belong to the background area. In the second next frame #n+2, the 
second to fourth pixels from left contain only the background components and belong 
to the background area. 

Since the object corresponding to the background is still, the pixel value of the 
second pixel from left of the frame #n+2 is not changed from the pixel value of the 
second pixel from left of tie frame #n+l . Similarly, the pixel value of the second pixel 
from left of the frame #n +2 is not changed from the pixel value of the second pixel 
from left of the frame #n- u l, whilst the pixel value of the third pixel from left of the 
frame #n+2 is not changed from the pixel value of the fourth pixel from left of the 
frame #n+l. 
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That is, the pixels of the frame #n+l and frame #n+2 corresponding to the 
pixels belonging to the uncovered background area in the frame #n are composed only 
of background components and are not changed in the pixel values. So> the absolute 
value of the difference is approximately zero. Therefore, the pixels of the frame #n-H 
and frame #n+2 corresponding to the pixels belonging to the mixed area in the frame 
#n are decided by the still/moving discriminating unit 122-1 to be still pixels. 

The pixels belonging to the uncovered background area in the frame #n contain 
the foreground components and hence differ in pixel values from the pixels in the 
frame #n+l composed only of the background components. So, the pixels belonging 
to the mixed area in the frame #n and those of the corresponding frame #n* 1 are 
decided by the still/moving discriminating unit 122-2 to be moving pixels. 

The area decision unit 123-1 is fed in this manner with the result indicating 
movement from the still/moving discriminating unit 122-2. If fed with the result 
indicating still from the slill/moving discriminating unit 122-1, the area decision unit 
123-1 decides that the con esponding pixel belongs to the uncovered background area. 

Fig. 3 3 shows decision conditions of the area specifying unit 103 in the frame 
#n. When the pixel of die frame #n*2 at the same position as the position on the 
picture of the pixel of the frame #n being verified and the pixel of the frame #n* 1 at 
the same position as the position on the picture of the pixel of the frame #n being 
verified, are decided to be still, whilst the pixel of the frame #n-l at the same position 
as the position on the picture of the pixel of the frame #n being verified and the pixel 
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of the frame #n are decided to be moving, the area specifying unit 103 decides that the 
pixel of the frame #n being verified belongs to the covered background area. 

When the pixel of the frame #n*l at the same position as the position on the 
picture of the pixel of the frame #n being verified and the pixel of the frame #n are 
decided to be still, whilsi: the pixel of the frame #n and the pixel of the frame #n+ 1 at 
the same position as the position on the picture of the pixel of the frame #n being 
verified are decided to be still, the area specifying unit 1 03 decides that the pixel of the 
frame #n being verified belongs to the still area. 

When the pixel of the frame #n l at the same position as the position on the 
picture of the pixel of the frame #n being verified and the pixel of the frame #n are 
decided to be moving, whilst the pixel of the frame #n and the pixel of the frame #n+l 
at the same position as the position on the picture of the pixel of the frame #n being 
verified are decided to be still, the area specifying unit 103 decides that the pixel of the 
frame #n being verified bslongs to the moving area. 

When the pixel of the frame #n and the pixel of the frame #n+l at the same 
position as the position on the picture of the pixel of the frame #n being verified are 
decided to be moving and when the pixel of the frame #n+l at the same position as the 
position on the picture of ihe pixel of the frame #n being verified and the pixel of the 
frame #n+ 1 at the same position as the position on the picture of the pixel of the frame 
#n being verified and the pixel of the frame #n+2 at the same position as the position 
on the picture of the pixel of the frame #n being verified are decided to be still, the 
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area specifying unit 103 decides that the pixel of the frame #n being verified belongs 
to the uncovered background area. 

Fig. 34 shows an example of the area decision by the area specifying unit 103. 
In Fig. 34 A, a pixel dee ded to belong to the covered background area is shown in 
white. In Fig.34B, a pixel decided to belong to the uncovered background area is 
shown in white. 

In Fig.34C, a pixel decided to belong to the moving area is shown in white. In 
Fig.34D, a pixel decided to belong to the still area is shown in white. 

Fig.3 5 shows the area information representing the mixed area, among the area 
information output by the decision flag storage frame memory 126, as picture. In 
Fig. 35, the pixel decided to belong to the covered background area or the uncovered 
background area, that is to the mixed area, is shown in white. The area information 
indicating the mixed area, output by the decision flag storage frame memory 126, 
indicates a textured portion surrounded by an untextured portion in the foreground 
area and the mixed area. 

Referring to the flowchart of Fig. 36, the processing for area identification by 
the area specifying unit 103 is explained. At step S121, the frame memory 121 
acquires pictures of the fl ame #n-2 to frame #n+2, inclusive the frame #n. 

At step S122, the still/moving discriminating unit 122-3 checks whether or not 
the pixels at the same po sition of the frame #n-l and the frame #n are still. If the 
pixels are decided to be still, the program moves to step S123 where the still/moving 
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discriminating unit 122-2 checks whether or not the pixels at the same position of the 
frame #n and the frame #n+l are still. 

If, at step S 123, tb e pixels at the same position of the frame #n and the pixel of 
the frame #n+l are decided to be still, the program moves to step SI 24 where the area 
decision unit 123-2 sets " 1" in the still area decision flag corresponding to the pixel of 
the area being verified for indicating that the pixel belongs to the still area. The area 
decision unit 123-2 sends the still area decision flag to the decision flag storage 
memory 124. The prognim then moves to step S125. 

If at step SI 22 the pixels at the same position of the frame #n*l and the frame 
#n are decided to be moving or if at step SI 23 the pixels at the same position of the 
frame #n and the frame #n+l are decided to be moving, the pixel of the frame #n does 
not belong to the still area, so the processing at stepS124 is skipped and the program 
moves to step S125. 

At step S 125, the si ill/moving discriminating unit 122-3 checks whether or not 
the pixels at the same position of the frame #n*l and the frame #n are moving. If the 
pixels are decided to be moving, the program moves to step S126 where the 
still/moving discriminating unit 122-2 decides whether or not the pixels at the same 
position of the frame #n and the frame #n+l are moving. 

If, at step S126, the pixels at the same position of the frame #n and the pixel of 
the frame #n+I are decided to be moving, the program moves to step SI 27 where the 
area decision unit 1 23-2 seis " 1 " in the movung area decision flag conresponding to the 
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pixel of the area being verified for indicating that the pixel belongs to the moving area. 

The area decision unit 123-2 sends the moving area decision flag to the decision flag 

storage memory 124. The program then moves to step S 128. 

If at step S125 tho pixels at the same position of the frame fori and the frame 
#n are decided to be still or if at step S 1 26 the pixels at the same position of the frame 
#n and the frame #n+l are decided to be still, the pixel of the frame #n does not belong 
to the moving area, so the processing at stepS127 is skipped and the program moves 
to stepS 128. 

At step S 128, the s till/moving mscriminating unit 122-4 checks whether or not 
the pixels at the same position of the frame #n-2 and the frame #n-l are still. If the 
pixels are decided to be still, the program moves to step SI 29 where the still/moving 
discrirninatingunit 122-3 decides whether or not the pixels at the same position of the 
frame #n* 1 and the frame #n are moving. 

If, at step S 129, the pixels at the same position of the frame #n- 1 and the pixel 
of the frame #n are decided to be moving, the program moves to step S130 where the 
area decision unit 123-3 sets "1" in the covered background area decision flag 
corresponding to the pixel of the area being verified for indicating that the pixel 
belongs to the covered background area. The area decision unit 123-3 sends the 
covered background area decision flag to the decision flag storage memory 124. The 
program then moves to step S131. 

If at step S 128 the pixels at the same position of the frame #n-2 and the frame 
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#n-2 are decided to be rr oving or if at step S129 the pixels at the same position of the 
frame #n- 1 and the frame #n are decided to be still, the pixel of the frame #n does not 
belong to the covered background area, so the processing at stepS130 is skipped and 
the program moves to stup SI 31. 

At step S 131, the still/moving discriminating unit 122-2 checks whether or not 
the pixels at the same position of the frame #n and the frame #n+l are still. If the 
pixels are decided to he moving, the program moves to step S132 where the 
still/moving discriminating unit 122-1 decides whether or not the pixels at the same 
position of the frame #n-f 1 and the frame #n+2 are moving. 

If, at step S132, the. pixels at the same position of the frame #n+l and the pixel 
of the frame #n+2 are decided to be still, the program moves to step S133 where the 
area decision unit 123-1 sets "1" in the uncovered background area decision flag 
corresponding to the pixd of the area being verified for indicating that the pixel 
belongs to the uncovered background area. The area decision unit 123-1 sends the 
uncovered background area decision flag to the decision flag storage memory 124. 
The program then moves 1o step S134. 

If at step S13 1 the pixels at the same position of the frame #n and the frame 
#n+l are decided to be still or if at step S132 the pixels at the same position of the 
frame #n+l and the frame #n+2 are decided to be moving, the pixel of the frame #n 
does not belong to the uncovered background area, so the processing at stepS133 is 
skipped and the program moves to step SI 34. 
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At step S134, the area specifying unit 103 checks whether or not the area has 
been specified for the totality of the pixels of the frame #n. If it is decided that the 
area has not been specified for the totality of the pixels of the frame #n, the program 
reverts to step S 122 to repeat the area specifying processing for the remaining pixels. 

If it is decided at step SI 34 that the area has been specified for the totality of 
the pixels of the frame fca, the program moves to step S135 where the synthesis unit 
1 25 generates the area information indicating the mixed area based on the uncovered 
background area decision flag and the covered background area decision flag, stored 
in the decision flag storage memory 124, while also generating the area information 
indicating to which of the uncovered background area, still area, moving area and the 
uncovered background area belongs each pixel. The synthesis unit 125 sets the 
generated area informatioa in the decision flag storage frame memory 126 to finish the 
processing. 

In this manner, the area specifying unit 103 is able to generate the area 
information, for each of pixels comprehended in a frame, indicating that the pixel in 
question belongs to the movement area, still area, covered background area or to the 
uncovered background area. 

It is also possible for the area specifying unit 103 to apply logical sum to area 
information corresponding to the uncovered background area and the covered 
background area to generate the area information comprising a flag indicating that a 
given pixel contained in the frame belongs to the movement area, still area or to the 
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mixed area, for each pixel contained in the frame. 

If the object associated with the foreground includes a texture, the area 
specifying unit 103 is able to specify the movement area more accurately. 

The area specifying unit 103 is able to output the area information indicating 
the movement area as the area information indicating the foreground area, or the area 
information indicating the still area as the area information indicating the background 
area. 

In the foregoing, it is assumed that the object corresponding to the background 
is still. However, the above-described area specifying processing can be applied even 
if the picture associated with the background area contains motion. For example, if 
the picture corresponding to the background area is moving uniformly, the area 
specifying unit 103 shifts the entire picture in association with the movement to 
perform the processing in the same way as when the object corresponding to the 
background is still. If the picture associated with the background area contains 
different movements from one location to another, the area specifying unit 103 selects 
the pixels corresponding to the movement to perform the above processing. 

Fig.37 shows a blosk diagram showing an illustrative structure of the mixing 
ratio calculating unit 104. The estimated mixing ratio processor 201 calculates the 
estimated mixingratio, from one pixel to another, by calculations corresponding to the 
model of the covered background area, based on the input picture, to route the 
calculated estimated mixin g ratio to a mixing ratio decision unit 203. 
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An estimated mixing ratio processing unit 202 calculates the estimated mixing 
ratio, from pixel to pixel by calculations corresponding to the model of the uncovered 
background area, based on the input picture, to route the calculated mixing ratio to the 
mixing ratio decision unit 203. 

Since the object corresponding to the foreground may be assumed to be moving 
at an equal speed within the shutter time, the mixing ratio a of a pixel belonging to the 
mixed area has the following properties: That is, the mixing ratio a is changed linearly 
relative to changes in the pixel positions. If the changes of the pixel positions are one- 
dimensional, the changes in the mixing ratio a can be represented as a plane. 

Since the one-frame period is short, it may be assumed that the object 
corresponding to the foreground is a rigid member and is moving at ah equal speed. 

Meanwhile, the tilt of the mixing ratio a is inversely proportionate to the 
movement quantity v of the foreground within the shutter time. 

Fig.38 shows an example of an ideal mixing ratio a. The tilt 1 in the mixing 
area with an ideal mixing ratio a can be represented as a reciprocal of the movement 
• quantity v. . 

In the embodiment Df Fig.39, the pixel value C06 of the seventh pixel from left 
of the frame #n can be represented, using the pixel value P06 of the seventh pixel from 
left of the frame #n-l, by the equation (14): 
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C06= B06/v+ B06/v+ F01/v+ F02/v 
= P06/v+ P06/v+ F01/v+ F02/v 

2 

= 2/vP06+£ F,/v 
1=1 

- (14) 

In the equation (14), the pixel value C06 is expressed as a pixel value M of the 
pixel of the mixed area, whilst the pixel value P06 is expressed as a pixel value B of 
the pixel of the background area. That is, the pixel value M of the mixed area and the 
pixel value B of the background may be represented by the equations (15) and (16), 
respectively: 

M = C06 

-(15) 

C = a-P+f 

- (16). 

In the equation (14), 2/v corresponds to the mixing ratio a. Since the 
movement quantity v is A, the mixing ratio a of the seventh pixel from left of the 
frame #n is 0.5. 

By assuming that th e pixel value C of the frame #n under consideration and the 
pixel value P of the frame #n- 1 directly previous to the frame #n as being the pixel 
value of the mixed area and the pixel value of the background, respectively, the 
equation (13) indicating the mixing ratio a can be rewritten to the following equation 
(17): 
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C = CC\P + f 

-(17) 

where f denotes the sum SjFi/v of the foreground components contained in the 
considered pixel. There ;ire two variables in the equation (17), namely the mixing ratio 
a and the sum f of the foreground components. 

Fig.40 shows a model in which the movement quantity v in the uncovered 
background area is 4 and the number of times of the virtual splitting along the rime 
axis is 4, with the pixels being shown developed along the time axis direction. 

By assuming, in the uncovered background area, that the pixel value C of the 
frame #n under consideration and the pixel value P of the frame #n+l next to the 
frame #n as being the pixel value of the mixed area and the pixel value of the 
background, respectively, as in the covered background area, discussed above, the 
equation (13) indicating ihe mixing ratio a can be represented as in the following 
equation (18): 

C = a-N + f 

— (18). 

Although the background object is assumed to be still in the foregoing 
description, the equations (14) to (18) may be applied by exploiting the pixel values 
of the pixels associated with the background movement quantity v even if the 
background object is moving. For example, if, when the movement quantity v of the 
object corresponding to the background is 2 and the number of times of the virtual 
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splitting is 2, the object corresponding to the background is moving towards right in 
the drawing, the pixel value B of the pixel of the background area in the equation (16) 
is the pixel value P04. 

Since the equation (17) and (18) each contain two variables, the mixing area a 
cannot be found directly. It should be noted that, since the picture in general exhibits 
strong spatial correlation, the pixels proximate to each other are of approximately the 
same pixel values. 

Since the foreground components exhibits strong spatial correlation, the 
equation is modified so that the mixing area a by the sum of the foreground 
components will be derived from the previous frame or the subsequent frames. 

The pixel value Mc of the seventh pixel from left of the frame #n of Fig. 4 1 can 
be represented by the following equation (19): 

2 12 

v ,tr, -(19) 

where 2/v of the first term of the right side corresponds to the mixing ratio a. By 
exploiting the pixel value of the subsequent frame #n-H, the second term of the right 
side of the equation (19) may be represented by the equation (20): 

t FUV - fi -l FUV -(20). 

It is here assumed, by exploiting the spatial correlation of the foreground 
components, the following equation (21) holds: 
F = F05 = F06 = F07 = F08 - F09 - F10 = Fl 1 = F12 
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-(21) 



which may be used to rewrite the equation (20) to 



2 

v 



- (22). 

As a result, p can be represented by the following equation (23) 
P=22/4 

- (23). 

In general, if it is assumed that the foreground components relevant to the 
mixed area are equal, as ,hown by the equation (21), the following equation (24): 
P = 1-a 

— (24) 

holds, by the ratio of the internal division, for the totality of pixels of the mixed area. 
If the equation (24) holds, the equation ( 1 7) can be expanded as in the equation 

(25): 



C=aP + f 



'v 



-(25). 

^ uation (24 ) holds, the equation (18) can be expanded as in 

the equation (26): 
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C=aN+f 
= a-N+(l-a)- }] Fi/v 
= a-N+(l-a)-P 

- (26). 

In the equations (25) and (26), since C, N and P are known pixel values, the 
there is only one variable contained in the equations (25) and (26), that is the mixing 
ratio a. The relation air on g C, N and P in the equations (25) and (26) is shown in 
Fig.42. It is noted that C, N and P are a pixel value of a pixel of the frame #n under 
consideration, a pixel value of a pixel of the frame #n+l, the position of which in the 
spatial direction is in register with that of the considered pixel, and a pixel value of the 
pixel of the frame #n+l , the position of which in the spatial direction is in register with 
that of the considered pixel, respectively. 

Thus, each one variable is contained in each of the equations (25) and (26), so 
the mixing ratio a can be calculated by exploiting the pixel values of the pixels of the 
three frames. The conditio for the correct mixing ratio a to be calculated by solving 
the equations (25) and (26) is that the foreground components relevant to the mixed 
area are equal, that is that nhe pixel values of a number of the consecutive pixels twice 
the movement quantity x, which pixels are in the picture object of the foreground 
imaged in a standstill sta te, and which are positioned at a boundary of the picture 
object in association with the moving direction of the foreground are constant. 

The mixing ratio Of of the pixels belonging to the covered background area is 
calculated by the equation (27), whilst the mixing ratio a of the pixel belonging to the 
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uncovered background area is calculated by the following equations (27) and (28): 
a = (ON)/(P-N) 

(27) 

a = (OP)/(N-P) 

•» (28). 

In Fig.43, which is a block diagram showing the structure of the estimated 
mixing ratio processor 20 1 , a frame memory 22 1 stores the input pictures on the frame 
basis, and feeds a frame, next to the frame being input as an input picture, to a frame 
memory 222 and to a mixing ratio calculating unit 223. 

The frame memoiy 222 stores the input pictures on the frame basis and routes 
a frame next following the frame being supplied from the frame memory 221 to the 
mixing ratio calculating unit 223. 

So, if the frame #n+l is being input as an input picture to the mixing ratio 
calculating unit 223, the frame memory 221 routes the frame #n to the mixing ratio 
calculating unit 223, whilst the frame memory 222 routes the frame #n* 1 to the mixing 
ratio calculating unit 223 . 

The mixing ratio cal culating unit 223 calculates an estimated mixing ratio of the 
considered pixel, by calcu] ations of the equation (27) on the pixel value C of the pixel 
of the frame #n under consideration, the pixel value of the pixel of the frame #n+l , the 
spatial position of which i s in registration with that of the considered pixel, and the 
pixel value of the pixel of the frame #n-l, the spatial position of which is in 
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registration with that of the considered pixel, and outputs the so-calculated estimated 
mixing ratio. For example, if the background is at a standstill, the mixing ratio 
calculating unit 223 calculates the estimated mixingratio of the considered pixel, from 
the pixel value C of the pixel of the frame #n under consideration, the pixel value N 
of the pixel of the frame #n+l, the position of which in the frame is the same as that 
of the considered pixel, and the pixel value P of the pixel of the frame #n-l, the 
position of which in the frame is the same as that of the considered pixel, and outputs 
the so-calculated estimated mixing ratio. 

In this manner, the estimated mixing ratio processor 201 calculates the 
estimated mixing ratio, based on the input picture, to route the so-calculated estimated 
mixing ratio to the mixing ratio decision unit 203. 

The estimated mbsng ratio processor 202 is similar to the estimated mixing 
ratio processor 201 exceptthatthe estimated mixing ratio processor 201 calculates the 
estimated mixing ratio of the considered pixel in accordance with the equation (27), 
whilst the estimated mixing ratio processor 202 calculates the estimated mixing ratio 
of the considered pixel in accordance with the equation (28), and hence the 
corresponding description is omitted for clarity. 

Fig.44 shows an example of the estimated mixing ratio calculated by the 
estimated mixing ratio processor 20 1 . Fig.44 shows the estimated mixing ratio for the 
movement quantity v of the foreground corresponding to an object moving at a 
constant speed equal to 1 1 for one line. 
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It is seen that the estimated mixing ratio is changing in the mixed area 
substantially linearly as shown in Fig.38. 

Reverting to Fig.:;7, the mixing ratio decision unit 203 sets the mixing ratio a 
based on the area information from the area specifying unit 103 indicating to which 
of the foreground area, background area, covered background area and the uncovered 
background area belongs the pixel supplied from the area specifying unit 103 as basis 
for calculation of the mi>ing ratio a. The mixing ratio decision unit 203 sets 0 or I as 
the mixing ratio if the pixel as a basis for calculation belongs to the foreground area 
or to the background area, respectively. On the other hand, the mixing ratio decision 
unit 203 sets the estimated mixing ratio supplied from the estimated mixing ratio 
processor 201 as the mixing ratio a if the pixel as a basis for calculation belongs to the 
covered background area, while setting the estimated mixing ratio supplied from the 
estimated mixing ratio pressor 202 as the mixing ratio a if the pixel as a basis for 
calculation belongs to the uncovered background area. The e203 outputs the mixing 
ratio a as set based on the area information. 

In Fig.45, which if; a block diagram showing an alternative structure of the 
mixing ratio calculating unit 1 04, a selection unit 23 1 routes the pixel belonging to the 
covered background area «ind pixels of the associated previous and subsequent frames 
to an estimated mixing raio processor 232, based on the area information supplied 
from the area specifying unit 103. The selection unit 23 1 routes the pixels belonging 
to the uncovered backgiound area and pixels of the associated previous and 
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subsequent frames to an estimated mixing ratio processor 233, based on the area 

information supplied from the area specifying unit 103. 

The estimated mixing ratio processor 232 calculates the estimated mixing ratio 

of the considered pixel belonging to the covered background area, by calculations in 

accordance with the equation (27), based on the pixel values input from the selection 
unit 231, to route the so-calculated estimated mixing ratio to a selection unit 234. 

The estimated mixing ratio processor 233 calculates the estimated mixing ratio 
of the considered pixel belonging to the uncovered background area, by calculations 
in accordance with the equation (28), based on the pixel values input from the 
selection unit 23 1 , to route the so-calculated estimated mixing ratio to a selection unit 
234. 

The selection unit 234 sets the mixing ratio a based on the area information 
from the area specifying unit 103 indicating to which of the foreground area, 
background area, covered background area and the uncovered background area 
belongs the pixel supplied from the area specifying unit 103 as basis for calculation 
of the mixing ratio a. The mixing ratio decision unit 203 sets 0 or 1 as the mixing ratio 
if the pixel as a basis for calculation belongs to the foreground area or to the 
background area, respectively. On the other hand, the selection unit 234 sets the 
estimated mixing ratio supplied from the estimated mixing ratio processor 232 as the 
mixing ratio a if the pixel £.s a basis for calculation belongs to the covered background 
area, while setting the estimated mixing ratio supplied from the estimated mixing ratio 
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processor 233 as the mixing ratio a if the pixel as a basis for calculation belongs to the 
uncovered background area. The selection unit 234 outputs the mixing ratio a 
selected and set based o n the area information. 

The mixing ratio calculating unit 104, having a modified structure shown in 
Fig.45, calculates the mixing ratio a, from one pixel of the picture to another, to 
output the calculated mi:dng ratio a. 

Referring to the flowchart of Fig.46, the processing for calculating the mixing 
ratio a of the mixing ratio calculating unit 104, the configuration of which is shown 
in Fig. 37, is explained. At step SI 51, the mixing ratio calculating unit 104 acquires 
the area information supplied from the area specifying unit 103. At step S151, the 
mixing ratio calculating unit 104 acquires the area information supplied from the area 
specifying unit 1 03. At step S 1 52, the estimated mixing ratio processor 20 1 calculates 
the estimated mixing ratio by a model corresponding to the covered background area 
to route the so-calculated estimated mixing ratio to the mixing ratio decision unit 203. 
The processing for calculating the estimated mixing ratio will be explained 
subsequently in detail by referring to flowchart of Fig.47. 

At step S 1 53, the es timated mixing ratio processor 202 calculates the estimated 
mixing ratio by a model corresponding to the covered background area to route the 
so-calculated estimated mixing ratio to the mixing ratio decision unit 203. 

At step SI 54, the mixing ratio calculating unit 104 checks whether or not the 
mixing ratio a has been estimated for the entire frame. If it is found that the mixing 
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ratio a has not been estimated for the entire frame, the program reverts to step SI 52 
to execute the processing of estimating the mixing ratio a for the next pixel. 

If it is decided at step SI 54 that the mixing ratio a has been estimated for the 
entire frame, the program reverts to step S 1 5 5 where the mixing ratio decision unit 203 
sets the mixing ratio a based on the area information supplied from the area specifying 
unit 103 and which indicates to which of the foreground area, background area, 
covered background area or the uncovered background area belongs the pixel. The 
mixing ratio decision uni t 203 sets 0 or 1 as the mixing ratio if the pixel as a basis for 
calculation belongs to the foreground area or to the background area, respectively. On 
the other hand, the mixing ratio decision unit 203 sets the estimated mixing ratio 
supplied from the estimated mixing ratio processor 201 as the mixing ratio a if the 
pixel as a basis for calcul ation belongs to the covered background area, while setting 
the estimated mixing ratio supplied from the estimated mixing ratio processor 202 as 
the mixing ratio a if the pixel as a basis for calculation belongs to the uncovered 
background area. The processing then is finished. 

In this manner, the mixing ratio calculating unit 104 is able to calculate the 
mixing ratio a, as a characteristic value for each pixel, based on the area information 
supplied from the area specifying unit 103 and on the input picture. 

The processing for calculating the mixing ratio a by the mixing ratio calculating 
unit 104 shown in Fig.45 is similar to that explained by referring to the flowchart of 
Fig.46 and hence is not explained specifically. 
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Referring to the flowchart of Fig.47, the processing similar to step SI 52 of 
Fig.46 for estimating the mixing ratio by a model corresponding to the covered 
background area is exphuned. 

At step SI71, the mixing ratio calculating unit 223 acquires the pixel value C 
of the considered pixel of the frame #n from the frame memory 221. 

At step S172, the nixing ratio calculating unit 223 acquires the pixel value C 
of the considered pixel of the frame #n-l from the frame memory 222. 

At step S173, the mixing ratio calculating unit 223 acquires the pixel value N 
of the frame #n+l, corresponding to the considered pixel contained in the input 
picture. 

At step S174, the mixing ratio calculating unit 223 calculates the estimated 
mixing ratio based on the pixel value C of the considered pixel of the frame #n, pixel 
value P of the pixel of ths frame #n-l and on the pixel value N of the pixel of the 
frame #n+l. 

At step S175, the mixing ratio calculating unit 223 checks whether or not the 
processing for calculating the estimated mixing ratio has been finished for the entire 
frame. If it is decided tha : the processing for calculating the estimated mixing ratio 
has not been finished forme entire frame, the program reverts to step S171 to repeat 
the processing of calculating the estimated mixing ratio for the next pixel. 

If it is verified at step SI 75 that the processing for calculating the estimated 
mixing ratio has been finished for the entire frame, the processing is finished. 
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In this manner, tb e estimated mixing ratio processor 20 1 is able to calculate the 
estimated mixing ratio based on the input picture. 

The processing for estimating the mixing ratio by the model corresponding to 
the uncovered background area at step SI 53 of Fig.46 is similar to the processing 
exploiting the equation corresponding to the model of the uncovered background area, 
as shown in the flowchart of Fig.47, and hence is not explained specifically. 

Meanwhile, since the estimated mixing ratio processor 232 and the estimated 
mixing ratio processor 233, shown in Fig.45, calculates the estimated mixing ratio by 
executing the processing similar to the processing of the flowchart of Fig.47, the 
corresponding operation is omitted for simplicity. 

In the foregoing explanation, it is assumed that the object corresponding to the 
background is at a standstill. However, the processing for finding the mixing ratio 0C 
can also be applied to a case in which a picture corresponding to the background 
contains the movement. For example, if a picture corresponding to the background 
area is moving uniformly, me estimated mixing ratio processor 201 shifts the entire 
picture in keeping with the movement of the background to perform the processing as 
if the object corresponding to the background is at a standstill. On the other hand, if 
the picture corresponding to the background area contains movements of the 
background which differ from one location to another, the estimated mixing ratio 
processor 20 1 selects the pixel associated with the background movement, as the pixel 
corresponding to the pixel belongingto the mixed area, to execute the above-described 
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processing. 

The structure of the mixing ratio calculating unit 104 shown in Figs. 37 or 45 is 
merely illustrative. 

It is also possible for the mixing ratio calculating unit 104 to execute only the 
processing for estimating the mixing ratio by the model corresponding to the covered 
background area to output the so-calculated estimated mixing ratio as the mixing ratio 
oc. In this case, the mixing ratio CL indicates the proportion of the foreground and the 
background for a pixel belonging to the covered background area and for a pixel 
belonging to the uncovered background area, respectively. If the absolute value of the 
difference between the so -calculated mixing ratio CL and 1 is calculated to set the so- 
calculated absolute value as the mixing ratio oc, the signal processor 12 is able to find 
the mixing ratio a indicating the proportion of the background component for the pixel 
belonging to the uncovered background area. 

It is also possible to execute only the processing for mixing ratio estimation by 
a model corresponding to the uncovered background area for the totality of the pixels 
to output the so-calculated estimated mixing ratio as the mixing ratio OC. 

The foreground/background separating unit 105 is now explained. In Fig.48 
which is a block diagram showing the illustrative structure of the 
foreground/background s eparating unit 105, an input picture, fed to the 
foreground/background separating unit 105, is supplied to a separating unit 251, a 
switch 252 and to a switch 254. The area information from the area specifying unit 
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